Electronic device including image sensor

ABSTRACT

An electronic device includes: a display; a first image sensor configured to output a first image signal based on sensing a first light passing through the display; a second image sensor configured to output a second image signal based on sensing a second light that does not pass through the display; and a processor configured to: generate a first optical value and a second optical value based on the second image signal, the second optical value being different from the first optical value; based on the first optical value satisfying a first condition, correct the first image signal by using the second optical value.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from Korean Patent Application No.10-2020-0097880, filed on Aug. 5, 2020 in the Korean IntellectualProperty Office, and all the benefits accruing therefrom under 35 U.S.C.119, the disclosure of which is incorporated by reference herein in itsentirety.

BACKGROUND 1. Field

The disclosure relates to an electronic device including an imagesensor. More particularly, the disclosure relates to an electronicdevice including a plurality of image sensors and an image signalprocessor.

2. Description of Related Art

An image sensing device is a semiconductor device that converts opticalinformation into an electrical signal. Examples of the image sensingdevice may include a charge coupled device (CCD) image sensing deviceand a complementary metal oxide semiconductor (CMOS) image sensingdevice.

A CMOS image sensor (CIS) may include a plurality of pixels arrangedtwo-dimensionally. Each of the plurality of pixels may include, forexample, a photodiode (PD). The photodiode may serve to convert incidentlight into an electrical signal.

In recent years, with the development of the computer industry and thetelecommunications industry, demands for image sensors with improvedperformance have been increased in various fields, such as digitalcameras, camcorders, smartphones, game devices, security cameras,medical micro cameras, robots, autonomous vehicles, drones, and thelike.

Recently, an image sensor disposed under a display panel of anelectronic device is provided. As the image sensor senses light passingthrough the display panel, information on the sensed light may bedifferent from information on actual light. For example, the quality ofan image outputted by the image sensor, which senses light passedthrough the display panel, may be deteriorated compared to the qualityof an image outputted by an image sensor which senses light not passedthrough the display panel. Therefore, a method for solving this problemis needed in the image sensor that is disposed under a display panel ofan electronic device.

SUMMARY

One or more example embodiments of the disclosure provide an electronicdevice including an image sensor and a processor, in which productreliability is improved by reducing deterioration in image quality.

One or more example embodiments of the disclosure also provide anelectronic device including an image sensor and a processor, in whichproduct reliability is improved by enhancing image quality.

However, aspects of the disclosure are not restricted to those set forthherein. The above and other aspects of the disclosure will become moreapparent to one of ordinary skill in the art to which the disclosurepertains by referencing the detailed description of the disclosure givenbelow.

According to an aspect of an example embodiment, there is provided is anelectronic device including: a display; a first image sensor configuredto output a first image signal based on sensing a first light passingthrough the display; a second image sensor configured to output a secondimage signal based on sensing a second light that does not pass throughthe display; and a processor configured to: generate a first opticalvalue and a second optical value based on the second image signal, thesecond optical value being different from the first optical value; basedon the first optical value satisfying a first condition, correct thefirst image signal by using the second optical value.

The first optical value may include at least one of a luminance value oran illuminance value generated based on the second image signal, and thesecond optical value may include a color temperature value generatedbased on the second image signal.

The processor may be further configured to, based on the first opticalvalue satisfying the first condition and further based on the colortemperature value being less than a first color temperature or beingequal to or greater than a second color temperature, correct the firstimage signal by using the color temperature value, the second colortemperature being higher than the first color temperature.

The processor may be further configured to generate a third opticalvalue and a fourth optical value based on the first image signal, thefourth optical value being different from the third optical value.

The third optical value may include at least one of a luminance value oran illuminance value generated based on the first image signal, and thefourth optical value may include a color temperature value generatedbased on the first image signal.

The first condition may be that the first optical value is equal to orgreater than a threshold value.

The processor may be further configured to, based on the first opticalvalue not satisfying the first condition, correct the first image signalby using an optical value generated based on the first image signal.

The optical value generated based on the first image signal may includea color temperature value generated based on the first image signal.

The first condition may be that a difference between the first opticalvalue and the third optical value is less than a threshold value.

The processor may be further configured to, based on the differencebetween the first optical value and the third optical value is equal toor greater than the threshold value, correct the first image signal byusing an optical value generated based on the first image signal.

The display may be further configured to output an image generated basedon the corrected first image signal.

The processor may be further configured to output a third image signalby performing auto white balance on the first image signal, and theprocessor may be further configured to, based on the first optical valuesatisfying a second condition, output a fourth image signal obtained bycorrecting the third image signal using the second optical value.

The display may be further configured to output an image generated basedon at least one of the third image signal and the fourth image signal.

According to an aspect of an example embodiment, there is provided anelectronic device, including: a first image sensor configured to outputa first image signal based on sensing a first light incident to a frontsurface of the electronic device; a second image sensor configured tooutput a second image signal based on sensing a second light incident toa rear surface of the electronic device; a processor configured toreceive the first image signal and the second image signal; and adisplay disposed on the front surface and configured to output an imagegenerated based on the first image signal, wherein the processor isfurther configured to generate a first color temperature value based onthe second image signal, and based on a first condition being satisfied,correct the first image signal by using the first color temperaturevalue.

The display may be further configured to cover the first image sensor.

The processor may be further configured to generate a second colortemperature value based on the first image signal, and generate at leastone of a luminance value or an illuminance value based on the secondimage signal.

The first condition may be that the at least one of the luminance valueor the illuminance value is equal to or greater than a threshold value.

The processor may be further configured to, based on the first conditionnot being satisfied, correct the first image signal by using the secondcolor temperature value.

The display may be further configured to output an image generated basedon the corrected first image signal.

According to an aspect of an example embodiment, there is provided anelectronic device including: a display; a first camera module includinga first image sensor, the first image sensor configured to output afirst image signal based on sensing a first light passing through thedisplay; a second camera module including a second image sensor, thesecond image sensor configured to output a second image signal based onsensing a second light that does not pass through the display; and anapplication processor disposed separately from the first camera moduleand the second camera module, and including an image signal processor,wherein the image signal processor is configured to: receive the firstimage signal from the first camera module through a first camera serialinterface; receive the second image signal from the second camera modulethrough a second camera serial interface; generate a first colortemperature value based on the first image signal; generate, based onthe second image signal, at least one of a luminance value or anilluminance value, and a second color temperature value; and correct thefirst image signal based on the second color temperature value based onthe at least one of the luminance value or the luminance value beingequal to or greater than a threshold value, and correct the first imagesignal based on the first color temperature value based on the at leastone of the luminance value or the illuminance value being less than thethreshold value.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certain exampleembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a perspective view illustrating an electronic device accordingto some example embodiments;

FIG. 2 is a partial cross-sectional view of the electronic device takenalong line A-A of FIG. 1;

FIG. 3 is a perspective view of the electronic device of FIG. 1 asviewed in a first direction;

FIG. 4 is a partial cross-sectional view of the electronic device takenalong line B-B of FIG. 3;

FIG. 5 is a block diagram illustrating an image sensing system accordingto some example embodiments;

FIG. 6 is a diagram for describing a conceptual layout of the imagesensor of FIG. 5;

FIG. 7 is a block diagram illustrating a first image signal processorand a second image signal processor of FIG. 5;

FIG. 8 is a block diagram illustrating a post-processing circuit of FIG.7;

FIG. 9 is a block diagram illustrating the post-processing circuit ofFIG. 8;

FIG. 10 is a flowchart illustrating an image compensation method by apost-processing circuit according to some example embodiments;

FIG. 11 is a block diagram illustrating the post-processing circuit ofFIG. 8;

FIG. 12 is a diagram illustrating image compensation conditionsaccording to some example embodiments;

FIG. 13 is a diagram describing an operation of an electronic deviceaccording to some example embodiments;

FIG. 14 is a block diagram illustrating an image sensing systemaccording to some other example embodiments;

FIG. 15 is a block diagram for describing an electronic device includinga multi-camera module according to some example embodiments; and

FIG. 16 is a detailed block diagram of the camera module of FIG. 15.

DETAILED DESCRIPTION

Hereinafter, example embodiments of the disclosure will be describedwith reference to the accompanying drawings.

Hereinafter, an electronic device including a first image sensor and asecond image sensor will be described with reference to FIGS. 1 to 4.

FIG. 1 is a perspective view illustrating an electronic device accordingto some example embodiments. FIG. 2 is a partial cross-sectional view ofthe electronic device taken along line A-A of FIG. 1. FIG. 3 is aperspective view of the electronic device of FIG. 1 as viewed in a firstdirection. FIG. 4 is a partial cross-sectional view of the electronicdevice taken along line B-B of FIG. 3.

Referring to FIGS. 1 to 4, an electronic device 1 may include a coverglass 10, a display 20, a rear glass 30, a back cover 40, a first imagesensor 100, a second image sensor 200, and the like.

In some example embodiments, the electronic device 1 may include ahousing. For example, the housing may include the cover glass 10 thatfaces a first direction D1 and the rear glass 30 that faces a fourthdirection D4 opposite to the first direction D1. Further, the housingmay include, for example, a connection part that connects the coverglass 10 to the rear glass 30. The housing may be used to protectcomponents inside the electronic device 1 from an external impact.

The cover glass 10 may include a transparent material so that electricalinformation signals displayed by the display 20 may be recognized fromthe outside. For example, the cover glass 10 may include glass orplastic.

The cover glass 10 and the rear glass 30 may have a flat shape. Forexample, each of the cover glass 10 and the rear glass 30 may have alength in a second direction D2 and a length in a third direction D3,which are greater than a length or thickness in the first direction D1.Accordingly, the electronic device 1 may have a flat shape. However,example embodiments according to the technical spirit of the disclosureare not limited thereto.

The surface of the electronic device 1 facing the first direction D1 maybe a front surface of the electronic device 1, and the surface of theelectronic device 1 facing the fourth direction D4 may be a rear surfaceof the electronic device 1. However, this is merely an example andexample embodiments according to the technical spirit of the disclosureare not limited.

Referring to FIGS. 1 and 2, the cover glass 10 may cover the display 20.For example, the display 20 may be disposed in the fourth direction D4from the cover glass 10.

The display 20 may include a plurality of pixels arranged along rows andcolumns. For example, the display 20 may include an organic lightemitting diode display (OLED), a liquid crystal display (LCD), a plasmadisplay panel (PDP), an electrochromic display (ECD), a digital mirrordevice (DMD), an actuated mirror device (AMD), a grating light value(GLV) device, an electroluminescent display (ELD), and the like.

The display 20 may output an image provided from the electronic device1. For example, signals are transmitted to the plurality of pixels andthe display 20 may output a corresponding image based on the signals.For example, the display 20 may include a touch screen or a touch panelthat displays images and/or receives an input by a user's touch.

The first image sensor 100 may sense light incident through the display20 from the outside of the electronic device 1 and/or a part of lightoutputted from the display 20. For example, the first image sensor 100may be covered by the display 20 and may sense light, which is incidentfrom the outside of the electronic device 1, in the electronic device 1.For example, the first image sensor 100 may sense light outputted fromthe display 20 and reflected by the cover glass 10.

For example, the first image sensor 100 may sense lights from the coverglass 10 and the pixels of the display 20, and light passed through agap between the pixels of the display 20.

Accordingly, a luminance value and a color temperature value of thelight passed through the display 20 may be changed by the cover glass 10and the pixels of the display 20. For example, the luminance value andthe color temperature value, which are generated as a result ofprocessing a signal outputted by sensing the light passed through thedisplay 20 by the first image sensor 100, may be smaller than the actualluminance value and color temperature value. However, embodiments of thedisclosure are not limited thereto.

Further, according to some example embodiments, the luminance value maybe used together with the illuminance value, or the illuminance valuemay be used instead of the luminance value. Hereinafter, in exampleembodiments according of the disclosure, it is described that theluminance value is used, but the illuminance value may also be used inthe same manner as the luminance value.

The first image sensor 100 may be covered by the display 20, andsurrounded by the back cover 40. For example, the first image sensor 100may be disposed in an opening formed in the back cover 40.

The back cover 40 may prevent light generated inside the electronicdevice 1 from affecting the display 20. Further, the back cover 40 mayprevent light outputted from the display 20 from entering the inside ofthe electronic device 1.

Although it is not shown in FIG. 2, the rear glass 30 may be disposed onthe back cover 40 and the first image sensor 100 in the fourth directionD4 and protect the back cover 40 and the first image sensor 100 from anexternal impact.

Referring to FIGS. 3 and 4, the rear glass 30 may be disposed at therear surface (e.g., the surface of the electronic device 1 in the fourthdirection D4) of the electronic device 1. Further, the second imagesensor 200 may be disposed at the rear surface (e.g., the surface of theelectronic device 1 in the fourth direction D4) of the electronic device1. However, embodiments of the disclosure are not limited thereto, andthe second image sensor 200 may be disposed at any other locations inthe electronic device 1, e.g., a side surface or the front surface ofthe electronic device 1.

The rear glass 30 may be disposed to surround the second image sensor200. For example, the second image sensor 200 may be disposed in anopening formed in the rear glass 30. A surface of the second imagesensor 200 may face the fourth direction D4. For example, the secondimage sensor 200 may have a light receiver that face the fourthdirection D4 (e.g., the direction toward the rear surface).

The second image sensor 200 may sense light incident on the rear surfaceof the electronic device 1. For example, the second image sensor 200 maysense light that does not pass through the display 20 disposed on thefront surface of the electronic device 1. That is, while the first imagesensor 100 senses the light passed through the display 20, the lightreceiver of the second image sensor 200 may sense the light that doesnot pass through the display 20 because the light receiver of the secondimage sensor 200 that faces the fourth direction D4 (e.g., the directiontoward the rear surface) it is not covered by the display 20 or anothercomponent of the electronic device 1.

Although one second image sensor 200 is illustrated, the second imagesensor 200 may include a plurality of image sensors. For example, thesecond image sensor 200 may include a regular camera, a wide-anglecamera, and a telephoto camera.

The rear glass 30 may prevent light from entering the electronic device1 from the outside. That is, it is possible to prevent light fromentering portions other than the opening of the rear glass 30 in whichthe second image sensor 200 is disposed.

Although it is not shown in FIG. 3, the cover glass 10, the display 20,and the back cover 40 may be disposed on the rear glass 30 in the firstdirection D1.

Hereinafter, an image sensing system 2 including the first image sensor100, the second image sensor 200, and an application processor 300 willbe described with reference to FIGS. 5 to 9.

FIG. 5 is a block diagram illustrating an image sensing system accordingto some example embodiments.

Referring to FIG. 5, the image sensing system 2 may include the firstimage sensor 100, the second image sensor 200, and the applicationprocessor 300.

The first image sensor 100 may generate a first image signal Si bysensing an image of a sensing target using incident light. For example,the first image sensor 100 may sense light passed through the display 20and generate the first image signal S1.

The second image sensor 200 may generate a second image signal S2 bysensing an image of a sensing target using incident light. For example,the second image sensor 200 may generate the second image signal S2 bysensing the light that does not pass through the display 20.

The first image signal S1 and the second image signal S2 may be providedto and processed by the application processor (AP) 300. For example, thefirst image signal S1 may be provided to and processed by a first imagesignal processor 310 of the AP 300, and the second image signal S2 maybe provided to and processed by a second image signal processor 330 ofthe AP 300. However, embodiments according of the disclosure are notlimited thereto. For example, the first image signal processor 310 andthe second image signal processor 330 may be implemented as one imagesignal processor.

The first image signal processor 310 may receive the first image signalS1 outputted from a buffer 170 of the first image sensor 100, andprocess the received first image signal S1 to be suitable for display.

The second image signal processor 330 may receive the second imagesignal S2 outputted from a buffer 270 of the second image sensor 200,and process the received second image signal S2 to be suitable fordisplay.

In some example embodiments, the first image signal processor 310 andthe second image signal processor 330 may perform digital binning on thefirst image signal S1 and the second image signal S2, respectively. Thefirst and second image signals S1 and S2 may respectively be raw imagesignals from pixel arrays 140 and 240, on which analog binning has notperformed, or may respectively be the first and second image signals S1and S2 on which the analog binning has been performed.

In some example embodiments, the first and second image sensors 100 and200, and the application processor 300 may be disposed to be separatefrom each other. For example, the first and second image sensors 100 and200 may be mounted on a first chip, and the application processor 300may be mounted on a second chip, and the first and second image sensors100 and 200 may communicate with the application processor 300 throughan interface. However, example embodiments of the disclosure are notlimited thereto. For example, the first and second image sensors 100 and200, and the application processor 300 may be implemented in onepackage, for example, a multi-chip package (MCP).

The first image sensor 100 may include a control register block 110, atiming generator 120, a row driver 130, the pixel array 140, a readoutcircuit 150, a ramp signal generator 160, and the buffer 170.

The control register block 110 may control the overall operation of theimage sensor 100. In particular, the control register block 110 maydirectly transmit operation signals to the timing generator 120, theramp signal generator 160 and the buffer 170.

The timing generator 120 may generate a reference signal for operationtiming of various components of the image sensor 100. The referencesignal for operation timing generated by the timing generator 120 may betransmitted to the row driver 130, the readout circuit 150, the rampsignal generator 160, and the like.

The ramp signal generator 160 may generate and transmit a ramp signalused in the readout circuit 150. For example, the readout circuit 150may include a correlated double sampler (CDS), a comparator, and thelike, and the ramp signal generator 160 may generate and transmit theramp signal used in the correlated double sampler (CDS), the comparator,and the like.

The buffer 170 may include, for example, a latch unit. The buffer 170may temporarily store the first image signal S1 to be provided to theoutside, and transmit the first image signal S1 to an external memory oran external device.

The pixel array 140 may sense an external image. The pixel array 140 mayinclude a plurality of pixels (or unit pixels). The row driver 130 mayselectively activate a row of the pixel array 140. The pixel array 140may acquire light incident through the display 20.

The readout circuit 150 may sample a pixel signal provided from thepixel array 140, compare the sampled pixel signal with the ramp signal,and convert an analog image signal (data) into a digital image signal(data) based on the comparison result.

FIG. 6 is a diagram for describing a conceptual layout of the imagesensor of FIG. 5.

Referring to FIG. 6, the first image sensor 100 may include a firstregion R1 and a second region R2 stacked in a first direction (e.g.,vertical direction). As illustrated, the first and second regions R1 andR2 may extend in a second direction (e.g., horizontal direction)intersecting the first direction, and a third direction intersecting thefirst and second directions. The blocks illustrated in FIG. 5 may bedisposed in the first and second regions R1 and R2.

Although not shown in the drawing, a third region in which a memory isarranged may be disposed on or under the second region R2. In this case,the memory disposed in the third region may receive image data from thefirst and second regions R1 and R2, store or process the image data, andretransmit the image data to the first and second regions R1 and R2. Inthis case, the memory may include a memory element, such as a dynamicrandom access memory (DRAM), a static random access memory (SRAM), aspin transfer torque magnetic random access memory (STT-MRAM), and aflash memory. When the memory includes, for example, a DRAM, the memorymay receive and process the image data at a relatively high speed.Further, in some example embodiments, the memory may be disposed in thesecond region R2.

The first region R1 may include a pixel array area PA and a firstperipheral area PH1, and the second region R2 may include a logiccircuit area LC and a second peripheral area PH2. The first and secondregions R1 and R2 may be sequentially stacked vertically.

In the first region R1, the pixel array area PA may be an area in whichthe pixel array (e.g., 140 of FIG. 5) is disposed. The pixel array areaPA may include a plurality of unit pixels arranged in a matrix. Eachpixel may include a photodiode and transistors.

The first peripheral area PH1 may include a plurality of pads and may bedisposed around the pixel array area PA. The plurality of pads maytransmit and/or receive electrical signals to and/or from the externaldevice.

In the second region R2, the logic circuit area LC may includeelectronic elements having a plurality of transistors. The electronicelements included in the logic circuit area LC may be electricallyconnected to the pixel array area PA to provide a signal to each unitpixel PX of the pixel array area PA or control an output signal of eachunit pixel PX.

In the logic circuit area LC, for example, the control register block110, the timing generator 120, the row driver 130, the readout circuit150, the ramp signal generator 160 and the buffer 170 described withreference to FIG. 5 may be disposed. For example, among the blocks ofFIG. 5, blocks other than the pixel array 140 may be disposed in thelogic circuit area LC.

Also, in the second region R2, the second peripheral area PH2 may bedisposed in an area corresponding to the first peripheral area PH1 ofthe first region R1, but embodiments are not limited thereto. The secondperipheral area PH2 may include a plurality of pads, which may transmitand/or receive electrical signals to and/or from the external device.

In the above, the first image sensor 100 has been described, but thesame or similar description may be also applied to the second imagesensor 200. For example, the configuration and operation of the controlregister block 110, the timing generator 120, the row driver 130, thepixel array 140, the readout circuit 150, the ramp signal generator 160and the buffer 170 of the first image sensor 100 may be substantiallythe same as or similar to the configuration and operation of a controlregister block 210, a timing generator 220, a row driver 230, the pixelarray 240, a readout circuit 250, a ramp signal generator 260 and thebuffer 270 of the second image sensor 200.

Further, in the logic circuit area LC of the second image sensor 200,one or more of the control register block 210, the timing generator 220,the row driver 230, the readout circuit 250, the ramp signal generator260, the buffer 270 and the like may be disposed.

However, there is a difference between the first image sensor 100 andthe second image sensor 200 in that the first image sensor 100 outputsthe first image signal S1, and the second image sensor 200 outputs thesecond image signal S2. Further, the pixel array 240 is different fromthe pixel array 140 in that the pixel array 240 may acquire light thatdoes not pass through the display 20.

FIG. 7 is a block diagram illustrating a first image signal processorand a second image signal processor of FIG. 5.

Referring to FIG. 7, the first image signal processor 310 may receivethe first image signal S1 from the first image sensor 100. The secondimage signal processor 330 may receive the second image signal S2 fromthe second image sensor 200.

The first image signal processor 310 may include a pre-processingcircuit 311, an image memory 317, a post-processing circuit 321, aconverting circuit 327, a data compressing circuit 328, and a memory329. The second image signal processor 330 may include a pre-processingcircuit 331, an image memory 337, a post-processing circuit 341, aconverting circuit 347, a data compressing circuit 348, and a memory349.

In some example embodiments, the pre-processing circuit 311 may performpre-processing on the first image signal S1. The pre-processing circuit311 may include a black level correction circuit 312, a defective pixelcorrection circuit 313, a shading correction circuit 314, an autoexposure (AE) evaluation value calculator 315, and an auto white balance(AWB) evaluation value calculator 316.

The first image signal S1 outputted from the first image sensor 100 maybe corrected so that the black level may become constant through blacklevel correction processing by the black level correction circuit 312.Information on the first image signal S1 may be interpolated by thedefective pixel correction circuit 313 based on information around adefective pixel when a pixel defect exists in the first image signal S1.Further, in the first image signal S1, a luminance difference betweenpixels, which is caused due to luminance omission occurring around apixel, may be corrected by the shading correction circuit 314.

The AE evaluation value calculator 315 may calculate an AE evaluationvalue AE1 based on a first conversion signal S1 a that has beensubjected to the corrections by the black level correction circuit 312,the defective pixel correction circuit 313, and the shading correctioncircuit 314. For example, the AE evaluation value calculator 315 maycalculate the AE evaluation value AE1 representing a brightness obtainedby integrating a luminance value sensed by the first image sensor 100.For example, the AE evaluation value AE1 may include the luminance valuesensed by the first image sensor 100.

The AWB evaluation value calculator 316 may calculate an AWB evaluationvalue AWB1 using a specific algorithm based on the AE evaluation valueAE1 and the first conversion signal S1 a that has been subjected to thecorrections by the black level correction circuit 312, the defectivepixel correction circuit 313, and the shading correction circuit 314.For example, the AWB evaluation value AWB1 may include a white balancegain to be used in white balance compensation processing. For example,the AWB evaluation value AWB1 may include a color temperature valuesensed by the first image sensor 100.

In this case, the color temperature value means that the color of thesensed light is represented as a temperature. For example, in the caseof red light, the color temperature value may be about 2,000 K, and inthe case of blue light, the color temperature value may be about 10,000K, but embodiments of the disclosure are not limited thereto.

The first conversion signal S1 a that has been subjected to thecorrections by the black level correction circuit 312, the defectivepixel correction circuit 313, and the shading correction circuit 314;the AE evaluation value AE1 outputted from the AE evaluation valuecalculator 315; and the AWB evaluation value AWB1 outputted from the AWBevaluation value calculator 316 may be temporarily stored in the imagememory 317 and transmitted to the post-processing circuit 321.

However, the embodiments of the disclosure are not limited thereto, andthe first conversion signal S1 a, the AE evaluation value AE1, and theAWB evaluation value AWB1 may be transmitted to the post-processingcircuit 321 without being stored in the image memory 317.

The post-processing circuit 321 may perform post-processing on the firstconversion signal S1 a which has been pre-processed by thepre-processing circuit 311. The post-processing circuit 321 may includea demosaicing processor 322, an edge emphasizing processor 323, a gammacompensating processor 324, a white balance compensating processor 325,a color compensating processor 326, and the like.

The demosaicing processor 322 may perform demosaic processing (e.g.,Bayer color interpolation processing) on the first conversion signal S1a transmitted from the pre-processing circuit 311. The edge emphasizingprocessor 323 may perform edge emphasizing processing on the firstconversion signal S1 a transmitted from the pre-processing circuit 311.The gamma compensating processor 324 may perform gamma compensation onthe first conversion signal S1 a transmitted from the pre-processingcircuit 311.

The white balance compensating processor 325 may receive the firstconversion signal S1 a and the AWB evaluation value AWB1 from thepre-processing circuit 311. The white balance compensating processor 325may perform white balance compensation processing on the firstconversion signal S1 a using the white balance gain of the AWBevaluation value AWB1.

The color compensating processor 326 may receive the first conversionsignal S1 a and the AWB evaluation value AWB1 from the pre-processingcircuit 311. The color compensating processor 326 may perform colorcompensation processing on the first conversion signal S1 a using thecolor temperature value sensed by the first image sensor 100 andincluded in the AWB evaluation value AWB1. Further, the colorcompensating processor 326 may perform color compensation processing onthe signal that has been subjected to the white balance compensationprocessing by using the color temperature value sensed by the firstimage sensor 100 and included in the AWB evaluation value AWB1.

However, embodiments of the disclosure are not limited thereto, and thecolor compensating processor 326 may perform the color compensation byusing other methods.

The first conversion signal S1 a that has been post-processed by thepost-processing circuit 321 may be transmitted to the converting circuit327. The converting circuit 327 may convert an RGB image signal into aYCbCr (YCC) image signal. Through the conversion processing, a colorspace of the captured image may be converted from a RGB color space to aYCC (YCrCb) color space.

The YCC image signal converted by the converting circuit 327 may betransmitted to the data compressing circuit 328. The data compressingcircuit 328 may compress the YCC image signal using a compressionformat, such as joint photographic experts group (JPEG). The compressedYCC image signal may be stored in the memory 329. Alternatively, thecompressed YCC image signal may be processed by the applicationprocessor 300 and the processed image may be outputted through thedisplay 20.

Although the first image signal processor 310 and the componentsincluded therein have been described as an example, the same or similardescription may be applied to the second image signal processor 330.That is, the configuration and operation of the pre-processing circuit311, the image memory 317, the post-processing circuit 321, theconverting circuit 327, the data compressing circuit 328, and the memory329 of the first image signal processor 310 may be the same as orsubstantially similar to the configuration and operation of apre-processing circuit 331, an image memory 337, a post-processingcircuit 341, a converting circuit 347, a data compressing circuit 348,and a memory 349 of the second image signal processor 330.

However, there is a difference between the first image signal processor310 and the second image signal processor 330 in that the first imagesignal processor 310 processes the first image signal S1 to generatedata, while the second image signal processor 330 processes the secondimage signal S2 to generate data. That is, the first image signalprocessor 310 may process the first image signal S1 outputted by sensingthe incident light passed through the display 20, while the second imagesignal processor 330 may process the second image signal S2 outputted bysensing the incident light without passing through the display 20.

For example, the pre-processing circuit 331 of the second image signalprocessor 330 may receive the second image signal S2.

An AE evaluation value calculator 335 of the pre-processing circuit 331may calculate an AE evaluation value AE2 based on a signal that has beensubjected to corrections by a black level correction circuit 332, adefective pixel correction circuit 333, and a shading correction circuit334. For example, the AE evaluation value calculator 335 may calculatethe AE evaluation value AE2 representing a brightness obtained byintegrating a luminance value sensed by the second image sensor 200. Forexample, the AE evaluation value AE2 may include the luminance valuesensed by the second image sensor 200.

The AE evaluation value calculator 335 may transmit the calculated AEevaluation value AE2 to the post-processing circuit 321 of the firstimage signal processor 310.

An AWB evaluation value calculator 336 may calculate an AWB evaluationvalue AWB2 using a specific algorithm based on the AE evaluation valueAE2 and the signal that has been subjected to the corrections by theblack level correction circuit 332, the defective pixel correctioncircuit 333, and the shading correction circuit 334. For example, theAWB evaluation value AWB2 may include a white balance gain to be used inwhite balance compensation processing. For example, the AWB evaluationvalue AWB2 may include a color temperature value sensed by the secondimage sensor 200.

The AWB evaluation value calculator 336 may transmit the calculated AWBevaluation value AWB2 to the post-processing circuit 321 of the firstimage signal processor 310.

In the drawing, the first image signal processor 310 and the secondimage signal processor 330 are illustrated as being separate from eachother, but this is for the purpose of description only and thedisclosure is not limited thereto. For example, the AE evaluation valuesAE1 and AE2, and the AWB evaluation values AWB1 and AWB2 may begenerated and shared by one processor (e.g., application processor 300).

Hereinafter, the post-processing circuit 321 of FIG. 7 will be describedin more detail with reference to FIG. 8.

FIG. 8 is a block diagram illustrating a post-processing circuit of FIG.7.

In some embodiments, the post-processing circuit 321 may include thewhite balance compensating processor 325 and the color compensatingprocessor 326.

The first conversion signal S1 a transmitted to the post-processingcircuit 321 may be the signal transmitted from the pre-processingcircuit 311, or a signal obtained by compensating the signal, which istransmitted from the pre-processing circuit 311, via the demosaicingprocessor 322, the edge emphasizing processor 323 and the gammacompensating processor 324.

The white balance compensating processor 325 may generate the secondconversion signal S1 b by performing the white balance compensationprocessing on the first conversion signal S1 a using the white balancegain of the AWB evaluation value AWB1. The white balance compensatingprocessor 325 may transmit the generated second conversion signal S1 bto a first determining unit 318.

The color compensating processor 326 may receive the second conversionsignal S1 b from the white balance compensating processor 325. However,embodiments according of the disclosure are not limited thereto, and thecolor compensating processor 326 may receive the first conversion signalS1 a, which is not white balance compensated, instead of the secondconversion signal S1 b.

The color compensating processor 326 may receive the AE evaluation valueAE2 and the AWB evaluation value AWB2 from the second image signalprocessor 330. For example, the color compensating processor 326 mayreceive, from the pre-processing circuit 331, the AE evaluation valueAE2 and the AWB evaluation value AWB2 of the signal that is sensed andoutputted by the second image sensor 200.

The color compensating processor 326 may generate a third conversionsignal S1 c by performing the color compensation processing based on atleast one of the received AE evaluation values AE1 and AE2, AWBevaluation values AWB1 and AWB2, and second conversion signal S1 b. Thecolor compensating processor 326 may transmit the generated thirdconversion signal S1 c to the first determining unit 318.

The first determining unit 318 may select and output at least one of thereceived second conversion signal S1 b and third conversion signal S1 c.For example, a fourth conversion signal S1 d may include at least one ofthe second conversion signal S1 b and the third conversion signal S1 c.The fourth conversion signal S1 d may be transmitted to other componentsof the application processor 300. For example, image data processedusing the fourth conversion signal S1 d may be outputted to the display20.

Hereinafter, the post-processing circuit 321 of FIG. 8 will be describedin more detail with reference to FIGS. 9 to 12.

FIG. 9 is a block diagram illustrating the post-processing circuit ofFIG. 8. FIG. 10 is a flowchart illustrating an image compensation methodby a post-processing circuit according to some example embodiments. FIG.11 is a block diagram illustrating the post-processing circuit of FIG.8. FIG. 12 is a diagram illustrating image compensation conditionsaccording to some example embodiments.

Referring to FIG. 9, the post-processing circuit 321 may further includea second determining unit 319. For example, the AWB evaluation valuesAWB1 and AWB2 may be transmitted to the second determining unit 319before transmitted to the color compensating processor 326. The AEevaluation value AE2 may also be transmitted to the second determiningunit 319.

In this case, the AE evaluation value AE1 may include the luminancevalue of light sensed by the first image sensor 100 after passingthrough the display 20, and the AE evaluation value AE2 may include theluminance value of light sensed by the second image sensor 200 withoutpassing through the display 20.

Further, the AWB evaluation value AWB1 may include the color temperaturevalue of light sensed by the first image sensor 100 after passingthrough the display 20, and the AWB evaluation value AWB2 may includethe color temperature value of light sensed by the second image sensor200 without passing through the display 20.

Referring to FIGS. 9 and 10, the post-processing circuit 321 may receivesignals from the first and second image sensors 100 and 200 (step S350).For example, the second determining unit 319 may receive the AEevaluation value AE2 and the AWB evaluation value AWB2 generated as theresult of processing the signal sensed by the second image sensor 200.For example, the second determining unit 319 may receive the AWBevaluation value AWB1 and the second conversion signal S1 b (or firstconversion signal S1 a) generated as the result of processing the signalsensed by the first image sensor 100.

The second determining unit 319 may determine whether the AE evaluationvalue AE2 is greater than a threshold value (step S351). In this case,the AE evaluation value AE2 may include the luminance value of lightsensed by the second image sensor 200 without passing through thedisplay 20. That is, when the environment where the electronic device 1is exposed is bright, the AE evaluation value AE2 may have a highervalue.

For example, referring to FIG. 12, in the case where the threshold valueis X1 lux, and the AE evaluation value AE2 is greater than X1 lux (Yesat step S351), the determining unit 319 may transfer the AWB evaluationvalue AWB2 to the color compensating processor 326 (step S352). That is,when the environment where the electronic device 1 is exposed issufficiently bright, the color compensating processor 326 may performthe color compensation using the AWB evaluation value AWB2 generated bysensing of the second image sensor 200 rather than using the AWBevaluation value AWB1 generated by sensing of the first image sensor100.

When the first image sensor 100 senses the light passed through thedisplay 20, information on the light may be distorted by the display 20or the like. Therefore, when the AE evaluation value AE2 is greater thanthe threshold value, the color compensating processor 326 may performthe color compensation using the AWB evaluation value AWB2 generated bysensing of the second image sensor 200. In an example embodiment, thecolor compensating processor 326 may perform the color compensationusing the AWB evaluation value AWB2 only when the AE evaluation valueAE2 is greater than the threshold value.

As the AWB evaluation value AWB2 is used, a more accurate colortemperature value may be used, thereby improving the image quality bythe color compensating processor 326.

Referring back to FIG. 10, if the AE evaluation value AE2 is not greaterthan the threshold value (No at step S351), the determining unit 319 maytransfer the AWB evaluation value AWB1 to the color compensatingprocessor 326 (step S353). That is, when the environment where theelectronic device 1 is exposed is not sufficiently bright, light sensedby the first image sensor 100 and light sensed by the second imagesensor 200 may be significantly different. Accordingly, in this case,the color compensation may be performed using the AWB evaluation valueAWB1 generated by sensing of the first image sensor 100.

After receiving one of the AWB evaluation values AWB1 and AWB2 from thedetermining unit 319, the color compensating processor 326 maycompensate the second conversion signal S1 b using the one of the AWBevaluation values AWB1 and AWB2.

The color compensating processor 326 may determine whether a colortemperature value derived from a corresponding one of the AWB evaluationvalues AWB1 and AWB2 is in a boundary region or not (step S354). Forexample, when the color compensating processor 326 receives the AWBevaluation value AWB1 from the determining unit 319, the colorcompensating processor 326 may determine whether a color temperaturevalue derived from the AWB evaluation value AWB1 is in the boundaryregion; and when the color compensating processor 326 receives the AWBevaluation value AWB2 from the determining unit 319, the colorcompensating processor 326 may determine whether a color temperaturevalue derived from the AWB evaluation value AWB2 is in the boundaryregion.

Referring to FIG. 12, when the color temperature value derived from thecorresponding one of the AWB evaluation values AWB1 and AWB2 is lessthan a first temperature T1 or greater than a second temperature T2, thecolor compensating processor 326 may compensate the second conversionsignal S1 b.

Referring back to FIG. 10, when the color temperature value derived fromthe corresponding one of the AWB evaluation values AWB1 and AWB2 is lessthan the first temperature T1 or greater than the second temperature T2(No at step S354), the color compensating processor 326 may perform thecolor compensation (step S355). For example, when the color temperaturevalue derived from the corresponding AWB evaluation value AWB1 or AWB2is less than about 3,000 K, the color compensating processor 326 maycompensate the second conversion signal S1 b with a redder color.Further, for example, when the color temperature value derived from thecorresponding AWB evaluation value AWB1 or AWB2 is greater than about10,000 K, the color compensating processor 326 may compensate the secondconversion signal S1 b with a bluer color. However, embodiments of thedisclosure are not limited thereto, and the color compensation method ofthe color compensating processor 326 may be performed in a differentmanner.

When the color temperature value derived from the corresponding one ofthe AWB evaluation values AWB1 and AWB2 is between the first temperatureT1 and the second temperature T2 (Yes at step S354), the colorcompensating processor 326 may complete the process without performingthe color compensation.

By causing the color compensating processor 326 to perform the colorcompensation on the second conversion signal S1 b, the quality of theimage corresponding to the second conversion signal S1 b may beimproved. Since the first image signal processor 310 performs the colorcompensation using the AWB evaluation value AWB2, which is generated bythe second image sensor 200 and the second image signal processor 330,only under a specific condition (for example, in an outdoor environmentwhich is sufficiently bright), accurate compensation may be performed.

Referring back to FIG. 9, the color compensating processor 326 mayoutput the third conversion signal S1 c which has been subjected to thecolor compensation. The third conversion signal S1 c may be transmittedto the first determining unit 318 of FIG. 8.

Referring to FIG. 11, the post-processing circuit 321 may furtherinclude a third determining unit 320. For example, the signal outputtedfrom the second determining unit 319 described with reference to FIGS. 9and 10 may be transmitted to the third determining unit 320. Further,the AWB evaluation value AWB1 may be transmitted to the thirddetermining unit 320.

The third determining unit 320 may determine which of the AWB evaluationvalue AWB1 and the AWB evaluation value AWB2 is to be transmitted to thecolor compensating processor 326, based on the difference between the AEevaluation values AE1 and AE2 (e.g., |AE1-AE2|).

For example, when the AE evaluation value AE2 is greater than the AEevaluation value AE1 by a certain difference or more (for example, whenthe luminance of light incident on the rear surface of the electronicdevice 1 is greater than the luminance of light incident on the frontsurface of the electronic device 1), the third determining unit 320 maytransmit the AE evaluation value AE1 to the color compensating processor326. Further, when the AE evaluation value AE2 is almost the same as theAE evaluation value AE1 (for example, the luminance of light incident onthe rear surface of the electronic device 1 is similar to the luminanceof light incident on the front surface of the electronic device 1), thethird determining unit 320 may transmit the AE evaluation value AE2received from the second determining unit 319 to the color compensatingprocessor 326.

Referring to FIG. 12, for example, when the difference between the AEevaluation value AE2 and the AE evaluation value AE1 is equal to orgreater than X2 lux, the third determining unit 320 may transmit the AEevaluation value AE1 to the color compensating processor 326.Alternatively, when the difference between the AE evaluation value AE2and the AE evaluation value AE1 is less than X2 lux, the thirddetermining unit 320 may transmit the AE evaluation value AE2 receivedfrom the second determining unit 319 to the color compensating processor326.

That is, when the difference between the luminance of light incident onthe front surface, which is measured by the first image sensor 100, andthe luminance of light incident on the rear surface, which is measuredby the second image sensor 200, is greater by a certain difference ormore, the color compensating processor 326 may perform the colorcompensation using the AE evaluation value AE1, rather than the AEevaluation value AE2. In this manner, color compensation may beperformed on the image by using the AE evaluation value AE2 uponsatisfaction of a preset condition.

FIG. 13 is a diagram describing an operation of an electronic deviceaccording to some example embodiments.

Referring to FIG. 13, signals may be transmitted between the first imagesensor 100, the second image sensor 200, the white balance compensatingprocessor 325, the color compensating processor 326, the applicationprocessor 300, and the display 20 that are included in the electronicdevice 1.

The first image sensor 100 may provide the AWB evaluation value AWB1,the AE evaluation value AE1, and the first conversion signal S1 a to thewhite balance compensating processor 325 (step S360). For example, theAWB evaluation value AWB1, the AE evaluation value AE1, and the firstconversion signal S1 a may be generated by the pre-processing circuit311 and transmitted to the white balance compensating processor 325 ofthe post-processing circuit 321.

The white balance compensating processor 325 may receive the AWBevaluation value AWB1, the AE evaluation value AE1, and the firstconversion signal S1 a, and perform the white balance compensation onthe first conversion signal S1 a based on the AWB evaluation value AWB1. Through this, the white balance compensating processor 325 maygenerate the second conversion signal S1 b (step S361).

The white balance compensating processor 325 may provide the generatedsecond conversion signal S1 b to the color compensating processor 326(step S362).

The first image sensor 100 may provide the AWB evaluation value AWB1 andthe AE evaluation value AE1 to the color compensating processor 326(step S363).

The second image sensor 200 may provide the AWB evaluation value AWB2and the AE evaluation value AE2 to the color compensating processor 326(step S364). For example, the AWB evaluation value AWB2 and the AEevaluation value AE2 may be generated by the pre-processing circuit 331and transmitted to the color compensating processor 326 of thepost-processing circuit 321.

The color compensating processor 326 may receive the AE evaluationvalues AE1 and AE2, the AWB evaluation values AWB1 and AWB2, and thesecond conversion signal S1 b (step S365).

The color compensating processor 326 may perform the color compensationon the second conversion signal S1 b based on the received signals (stepS366). For example, the color compensation processing of the colorcompensating processor 326, which is described with reference to FIGS. 8to 12, may be performed. Through this, the color compensating processor326 may generate the third conversion signal S1 c.

The color compensating processor 326 may provide the third conversionsignal S1 c to the application processor 300 (step S367). For example,the application processor 300 may receive the third conversion signal S1c. For example, a display driving circuit included in the applicationprocessor 300 may generate an image signal using the third conversionsignal S1 c.

The application processor 300 may provide the generated image signal tothe display 20 (step S368). For example, the image signal generated bythe display driving circuit may be transmitted to the display 20. As aresult, the display 20 may output an image based on the image signal.

Hereinafter, an image sensing system according to some other exampleembodiments will be described with reference to FIG. 14.

FIG. 14 is a block diagram illustrating an image sensing systemaccording to some other embodiments. For simplicity of description, adescription overlapping with the description with reference to FIGS. 1to 13 will be briefly given or omitted.

Referring to FIG. 14, an image sensing system 3 may include the firstimage sensor 100, the second image sensor 200, and the applicationprocessor 300.

The first image sensor 100 may generate the first image signal S1 bysensing an image of a sensing target using incident light. For example,the first image sensor 100 may sense light passed through the display 20and generate the first image signal S1.

The second image sensor 200 may generate the second image signal S2 bysensing the image of a sensing target using incident light. For example,the second image sensor 200 may generate the second image signal S2 bysensing light that does not pass through the display 20.

The first and second image signals S1 and S2 may be provided to andprocessed by an application processor 300. For example, the first andsecond image signals S1 and S2 may be provided to and processed by animage signal processor 390 included in the application processor 300.

The image signal processor 390 may process the first image signal S1 andthe second image signal S2 as described with reference to FIGS. 5 to 13.

For example, when processing the first image signal S1, the signalgenerated as the result of processing the second image signal S2 may beused under certain conditions. That is, when processing the first imagesignal S1, the signal generated by processing the second image signal S2may be shared in the image signal processor 390. Further, whenprocessing the second image signal S2, the signal generated byprocessing the first image signal S1 may be shared in the image signalprocessor 390.

That is, the processing steps performed by the first image signalprocessor 310 and the second image signal processor 330 of the imagesensing system 2 described with reference to FIGS. 5 to 13 may beperformed by the image signal processor 390 of the image sensing system3 of FIG. 14.

In this case, the shared signals are not limited to the signalsgenerated based on the first image signal S1 and the second image signalS2, and may include various other information. For example, the sharedsignals may include a signal for high dynamic range (HDR) processing ona plurality of image data.

Hereinafter, an electronic device according to some other embodimentswill be described with reference to FIGS. 15 and 16.

FIG. 15 is a block diagram for describing an electronic device 1000including a multi-camera module according to some embodiments. FIG. 16is a detailed block diagram of the camera module of FIG. 15. Forsimplicity of description, a description overlapping with thedescription with reference to FIGS. 1 to 14 will be briefly given oromitted.

Referring to FIG. 15, an image sensing system 4 includes the electronicdevice 1000, and the electronic device 1000 may include a camera modulegroup 1100, an application processor 1200, a power management integratedcircuit (PMIC) 1300, an external memory 1400, and a display 1500.

The camera module group 1100 may include a plurality of camera modules1100 a, 1100 b, and 1100 c. Although the drawing illustrates anembodiment in which three camera modules 1100 a, 1100 b, and 1100 c aredisposed, embodiments are not limited thereto. In some embodiments, thecamera module group 1100 may be modified to include only two cameramodules. In addition, in some embodiments, the camera module group 1100may be modified to include n (n is a natural number greater than 3)camera modules.

In this embodiment, one of the three camera modules 1100 a, 1100 b, and1100 c may be a camera module including the first image sensor 100described with reference to FIGS. 1 to 14. For example, the cameramodule 1100 b may be a camera module including the first image sensor100 and may be disposed toward the front surface of the electronicdevice 1 or 4. In addition, the camera module 1100 b may be covered bythe display 20 or the display 1500, and may sense incident light throughthe display 20 or 1500.

Hereinafter, a detailed configuration of the camera module 1100 b willbe described with reference to FIG. 16. The following description may beequally applied to other camera modules 1100 a and 1100 c according toembodiments.

Referring to FIG. 16, the camera module 1100 b may include a prism 1105,an optical path folding element (hereinafter, referred to as “OPFE”)1110, an actuator 1130, an image sensing unit 1140, and a storage unit1150.

The prism 1105 may include a reflective surface 1107 having a lightreflecting material and change the path of light L incident from theoutside.

In some embodiments, the prism 1105 may change the path of the light Lincident in a first direction X to a second direction Y perpendicular tothe first direction X. Further, the prism 1105 may rotate the reflectivesurface 1107 having the light reflecting material with respect to acentral axis 1106 in an A direction, or rotate the central axis 1106 ina B direction, thereby changing the path of the light L incident in thefirst direction X to the second direction Y perpendicular thereto. Inthis case, the OPFE 1110 may also move in a third direction Zperpendicular to the first direction X and the second direction Y.

In some embodiments, as shown in the drawing, the maximum rotation angleof the prism 1105 in the A direction may be 15 degrees or less in thepositive (+) A direction and greater than 15 degrees in the negative (−)A direction. However, embodiments are not limited thereto.

In some embodiments, the prism 1105 may move about 20 degrees, between10 and 20 degrees, or between 15 and 20 degrees in the positive (+) ornegative (−) B direction. In this case, the moving angle may be the samein the positive (+) or negative (−) B direction or may be almost thesame in the positive (+) or negative (−) B directions with a differenceof about 1 degree.

In some embodiments, the prism 1105 may move the reflective surface 1107having the light reflecting material in the third direction (e.g.,direction Z) parallel to the extending direction of the central axis1106.

The OPFE 1110 may include, for example, m (m is a natural number)optical lenses. The m lenses may move in the second direction Y tochange an optical zoom ratio of the camera module 1100 b. For example,if it is assumed that a basic optical zoom ratio of the camera module1100 b is Z, the optical zoom ratio of the camera module 1100 b maybecome 3 Z, 5 Z or more as the m optical lenses included in the OPFE1110 are moved.

The actuator 1130 may move the optical lenses or the OPFE 1110(hereinafter, referred to as “optical lens”) to a specific position. Forexample, the actuator 1130 may adjust the position of the optical lensso that an image sensor 1142 may be positioned at a focal length of theoptical lens for accurate sensing.

The image sensing unit 1140 may include the image sensor 1142, a controllogic 1144, and a memory 1146. The image sensor 1142 may sense an imageof a sensing target using the light L provided through the optical lens.In some embodiments, the image sensor 1142 may include at least one ofthe image sensors 100 and 200 described above.

The control logic 1144 may control the overall operation of the cameramodule 1100 b. For example, the control logic 1144 may control theoperation of the camera module 1100 b according to a control signalprovided through a control signal line CSLb.

The memory 1146 may store information such as calibration data 1147 thatis used for the operation of the camera module 1100 b. The calibrationdata 1147 may include information used for the camera module 1100 b togenerate image data using the light L provided from the outside. Thecalibration data 1147 may include, for example, information on anoptical axis, information on the degree of rotation, and information onthe focal length described above. When the camera module 1100 b isimplemented in the form of a multi-state camera whose focal lengthvaries according to the position of the optical lens, the calibrationdata 1147 may include information on auto focusing and a focal lengthvalue for each position (or state) of the optical lens.

The storage unit 1150 may store image data sensed through the imagesensor 1142. The storage unit 1150 may be disposed outside the imagesensing unit 1140, and may be implemented in a form of being stackedwith a sensor chip constituting the image sensing unit 1140. In someembodiments, the storage unit 1150 may be implemented as an ElectricallyErasable Programmable Read-Only Memory (EEPROM), but embodiments are notlimited thereto.

Referring to FIGS. 15 and 16 together, in some embodiments, each of thecamera modules 1100 a, 1100 b, and 1100 c may include the actuator 1130.Accordingly, the camera modules 1100 a, 1100 b, and 1100 c mayrespectively include the calibration data 1147 that are the same ordifferent according to the operation of the actuators 1130 includedtherein.

In some embodiments, one camera module (e.g., camera module 1100 b)among the camera modules 1100 a, 1100 b and 1100 c may be a cameramodule of a folded lens type including the prism 1105 and the OPFE 1110described above, and the remaining camera modules (e.g., camera modules1100 a and 1100 c) may be camera modules of a vertical type, which donot include the prism 1105 and the OPFE 1110. However, embodiments arenot limited thereto.

In some embodiments, one camera module (e.g., camera module 1100 c)among the camera modules 1100 a, 1100 b and 1100 c may be a depth cameraof a vertical type that extracts depth information using, for example,infrared rays (IR). In this case, the application processor 1200 maymerge image data provided from the depth camera with image data providedfrom another camera module (e.g., camera module 1100 a or 1100 b) togenerate a three dimensional (3D) depth image.

In some embodiments, among the camera modules 1100 a, 1100 b, and 1100c, at least two camera modules (e.g., camera modules 1100 a and 1100 c)may have different fields of view (viewing angles). In this case, amongthe camera modules 1100 a, 1100 b, and 1100 c, for example, at least twocamera modules (e.g., camera modules 1100 a and 1100 c) may havedifferent optical lenses, but are not limited thereto.

Further, in some embodiments, the camera modules 1100 a, 1100 b, and1100 c may have different viewing angles. In this case, optical lensesincluded in the respective camera modules 1100 a, 1100 b, and 1100 c mayalso be different, but the disclosure is not limited thereto.

In some embodiments, the camera modules 1100 a, 1100 b, and 1100 c maybe disposed to be physically separate from each other. That is, thesensing area of one image sensor 1142 is not divided and used by all ofthe camera modules 1100 a, 1100 b, and 1100 c, but an independent imagesensor 1142 may be disposed inside each of the camera modules 1100 a,1100 b, and 1100 c.

In some embodiments, for example, the image sensor 1142 included in thecamera module 1100 a may have the image sensor 200 described above, andthe image sensor 1142 included in the camera module 1100 b may have theimage sensor 100 described above.

Referring back to FIG. 15, the application processor 1200 may include animage processing unit 1210, a memory controller 1220, and an internalmemory 1230. The application processor 1200 may be implementedseparately from the camera modules 1100 a, 1100 b, and 1100 c. Forexample, the application processor 1200 and the camera modules 1100 a,1100 b, and 1100 c may be implemented separately as separatesemiconductor chips.

The image processing unit 1210 may include a plurality of sub-imageprocessors 1212 a, 1212 b, and 1212 c, an image generator 1214, and acamera module controller 1216. In some embodiments, the image processingunit 1210 may include the first image signal processor 310, the secondimage signal processor 330, and the image signal processor 390 describedabove.

The image processing unit 1210 may include the sub-image processors 1212a, 1212 b, and 1212 c, the number of which corresponds to the number ofthe camera modules 1100 a, 1100 b, and 1100 c. In some embodiments, eachof the sub-image processors 1212 a, 1212 b, and 1212 c may include oneof the first image signal processor 310, the second image signalprocessor 330, and the image signal processor 390 described above.

Image data generated from the respective camera modules 1100 a, 1100 b,and 1100 c may be provided to the corresponding sub-image processors1212 a, 1212 b, and 1212 c through separate image signal lines ISLa,ISLb, and ISLc. For example, image data generated from the camera module1100 a may be provided to the sub-image processor 1212 a through theimage signal line ISLa, image data generated from the camera module 1100b may be provided to the sub-image processor 1212 b through the imagesignal line ISLb, and image data generated from the camera module 1100 cmay be provided to the sub-image processor 1212 c through the imagesignal line ISLc. For example, such image data transmission may beperformed using a camera serial interface (CSI) based on a mobileindustry processor interface (MIPI), but embodiments are not limitedthereto.

In some embodiments, one sub-image processor may be arranged tocorrespond to a plurality of camera modules. For example, the sub-imageprocessor 1212 a and the sub-image processor 1212 c may be integratedinto one sub-image processor without being separate from each other asshown, and image data provided from the camera module 1100 a and thecamera module 1100 c may be selected by a data selector (e.g.,multiplexer) or the like, and then provided to the integrated sub-imageprocessor.

The image data provided to each of the sub-image processors 1212 a, 1212b, and 1212 c may be provided to the image generator 1214. The imagegenerator 1214 may generate an output image using the image dataprovided from each of the sub-image processors 1212 a, 1212 b, and 1212c according to image generation information or a mode signal. In someembodiments, the image generator 1214 may include the post-processingcircuit 321 described above. The image generator 1214 may perform theabove-described compensation process.

Specifically, according to the image generation information or the modesignal, the image generator 1214 may generate an output image by mergingat least some of the image data generated from the camera modules 1100a, 1100 b and 1100 c having different viewing angles. Further, accordingto the image generation information or the mode signal, the imagegenerator 1214 may generate an output image by selecting any one ofimage data generated from camera modules 1100 a, 1100 b and 1100 chaving different viewing angles.

In some embodiments, the image generation information may include a zoomsignal (or zoom factor). Further, in some embodiments, the mode signalmay be a signal based on, for example, a mode selected by a user.

When the image generation information is a zoom signal (zoom factor),and the camera modules 1100 a, 1100 b, and 1100 c have different fieldsof view (viewing angles), the image generator 1214 may perform adifferent operation depending on the type of the zoom signal. Forexample, when the zoom signal is a first signal, image data outputtedfrom the camera module 1100 a may be merged with image data outputtedfrom the camera module 1100 c, and then an output image may be generatedusing the merged image signal and image data that is outputted from thecamera module 1100 b that is not used for merging. When the zoom signalis a second signal that is different from the first signal, the imagegenerator 1214 does not perform the image data merging, but may generatean output image by selecting any one of image data outputted from thecamera module 1100 a, 1100 b, and 1100 c. However, embodiments are notlimited thereto, and the method of processing image data may bevariously modified as needed.

In some embodiments, the image generator 1214 may receive a plurality ofimage data having different exposure times from at least one of thesub-image processors 1212 a, 1212 b, and 1212 c, and perform the highdynamic range (HDR) processing on the plurality of image data, therebygenerating merged image data with an increased dynamic range.

The camera module controller 1216 may provide a control signal to eachof the camera modules 1100 a, 1100 b, and 1100 c. The control signalsgenerated from the camera module controller 1216 may be provided to thecorresponding camera modules 1100 a, 1100 b, and 1100 c through separatecontrol signal lines CSLa, CSLb, and CSLc.

Any one (e.g., camera module 1100 a) of the camera modules 1100 a, 1100b and 1100 c may be designated as a master camera according to the modesignal or the image generation information including the zoom signal,and the remaining camera modules (e.g., camera modules 1100 b and 1100c) may be designated as slave cameras. Such information may be includedin the control signals to be provided to the corresponding cameramodules 1100 a, 1100 b, and 1100 c through the separate control signallines CSLa, CSLb, and CSLc.

The camera modules operating as the master and the slaves may be changedaccording to the zoom factor or the operation mode signal. For example,when the viewing angle of the camera module 1100 a is wider than theviewing angle of the camera module 1100 c and the zoom factor indicatesa low zoom ratio, the camera module 1100 c may operate as a master, andthe camera module 1100 a may operate as a slave. On the contrary, whenthe zoom factor indicates a high zoom ratio, the camera module 1100 amay operate as a master, and the camera module 1100 c may operate as aslave.

In some embodiments, the control signal provided from the camera modulecontroller 1216 to each of the camera modules 1100 a, 1100 b, and 1100 cmay include a sync enable signal. For example, when the camera module1100 b is a master camera and the camera modules 1100 a and 1100 c areslave cameras, the camera module controller 1216 may transmit the syncenable signal to the camera module 1100 b. The camera module 1100 bhaving received the sync enable signal may generate a sync signal basedon the received sync enable signal, and transmit the generated syncsignal to the camera modules 1100 a and 1100 c through sync signal linesSSL. Based on the sync signal, the camera modules 1100 a, 1100 b and1100 c may synchronously transmit image data to the applicationprocessor 1200.

In some embodiments, the control signal provided from the camera modulecontroller 1216 to the camera modules 1100 a, 1100 b, and 1100 c mayinclude mode information according to the mode signal. Based on the modeinformation, the camera modules 1100 a, 1100 b, and 1100 c may operatein a first operation mode or a second operation mode in relation to asensing rate.

In the first operation mode, the camera modules 1100 a, 1100 b, and 1100c may generate an image signal at a first rate (e.g., first frame rate),and encode the image signal at a second rate (e.g., second frame ratehigher than first frame rate) higher than the first rate, and transmitthe encoded image signal to the application processor 1200. In thiscase, the second rate may be 30 times or less the first rate.

The application processor 1200 may store the received image signal, thatis, the encoded image signal, in the memory 1230 provided therein or ina storage 1400 provided outside the application processor 1200. Then,the application processor 1200 may read out the encoded image signalfrom the memory 1230 or the storage 1400 to decode the encoded imagesignal, and display image data generated based on the decoded imagesignal. For example, a corresponding sub-processor among the sub-imageprocessors 1212 a, 1212 b, and 1212 c of the image processing unit 1210may perform decoding and may also perform image processing on thedecoded image signal. For example, the image data generated based on thedecoded image signal may be displayed on the display 1500.

In the second operation mode, the camera modules 1100 a, 1100 b, and1100 c may generate an image signal at a third rate (e.g., third framerate lower than first frame rate) lower than the first rate and transmitthe image signal to the application processor 1200. The image signalprovided to the application processor 1200 may be an unencoded signal.The application processor 1200 may perform image processing on thereceived image signal or store the image signal in the memory 1230 orthe storage 1400.

The PMIC 1300 may supply power, such as a source voltage, to each of thecamera modules 1100 a, 1100 b, and 1100 c. For example, the PMIC 1300may supply first power to the camera module 1100 a through a powersignal line PSLa, second power to the camera module 1100 b through apower signal line PSLb and third power to the camera module 1100 cthrough a power signal line PSLc under the control of the applicationprocessor 1200.

The PMIC 1300 may generate power corresponding to each of the cameramodules 1100 a, 1100 b, and 1100 c in response to a power control signalPCON from the application processor 1200, and may also adjust a powerlevel. The power control signal PCON may include a power adjustmentsignal for each operation mode of the camera modules 1100 a, 1100 b, and1100 c. For example, the operation mode may include a low power mode,and in this case, the power control signal PCON may include informationon a camera module to operate in the low power mode and a set powerlevel. Levels of powers provided to the respective camera modules 1100a, 1100 b, and 1100 c may be the same or different. Further, the levelsof powers may be dynamically changed.

The disclosure may be implemented as a computer-readable code written ona computer-readable recording medium. The computer-readable recordingmedium may be any type of recording device in which data is stored in acomputer-readable manner.

Examples of the computer-readable recording medium include a ROM, a RAM,a CD-ROM, a magnetic tape, a floppy disc, an optical data storage, and acarrier wave (e.g., data transmission through the Internet). Thecomputer-readable recording medium can be distributed over a pluralityof computer systems connected to a network so that a computer-readablecode is written thereto and executed therefrom in a decentralizedmanner. Functional programs, codes, and code segments for implementingthe disclosure may be easily deduced by programmers of ordinary skill inthe art.

At least one of the components, elements, modules or units describedherein may be embodied as various numbers of hardware, software and/orfirmware structures that execute respective functions described above,according to an example embodiment. For example, at least one of thesecomponents, elements or units may use a direct circuit structure, suchas a memory, a processor, a logic circuit, a look-up table, etc. thatmay execute the respective functions through controls of one or moremicroprocessors or other control apparatuses. Also, at least one ofthese components, elements or units may be specifically embodied by amodule, a program, or a part of code, which contains one or moreexecutable instructions for performing specified logic functions, andexecuted by one or more microprocessors or other control apparatuses.Also, at least one of these components, elements or units may furtherinclude or implemented by a processor such as a central processing unit(CPU) that performs the respective functions, a microprocessor, or thelike. Two or more of these components, elements or units may be combinedinto one single component, element or unit which performs all operationsor functions of the combined two or more components, elements of units.Also, at least part of functions of at least one of these components,elements or units may be performed by another of these components,element or units. Further, although a bus is not illustrated in theblock diagrams, communication between the components, elements or unitsmay be performed through the bus. Functional aspects of the aboveexample embodiments may be implemented in algorithms that execute on oneor more processors. Furthermore, the components, elements or unitsrepresented by a block or processing operations may employ any number ofrelated art techniques for electronics configuration, signal processingand/or control, data processing and the like.

While the disclosure has been particularly shown and described withreference to embodiments thereof, it will be understood that variouschanges in form and details may be made therein without departing fromthe spirit and scope of the following claims.

What is claimed is:
 1. An electronic device comprising: a display; afirst image sensor configured to output a first image signal based onsensing a first light passing through the display; a second image sensorconfigured to output a second image signal based on sensing a secondlight that does not pass through the display; and a processor configuredto: generate a first optical value and a second optical value based onthe second image signal, the second optical value being different fromthe first optical value, and based on the first optical value satisfyinga first condition, correct the first image signal by using the secondoptical value.
 2. The electronic device of claim 1, wherein the firstoptical value includes at least one of a luminance value or anilluminance value generated based on the second image signal, andwherein the second optical value includes a color temperature valuegenerated based on the second image signal.
 3. The electronic device ofclaim 2, wherein the processor is further configured to, based on thefirst optical value satisfying the first condition and further based onthe color temperature value being less than a first color temperature orbeing equal to or greater than a second color temperature, correct thefirst image signal by using the color temperature value, the secondcolor temperature being higher than the first color temperature.
 4. Theelectronic device of claim 1, wherein the processor is furtherconfigured to generate a third optical value and a fourth optical valuebased on the first image signal, the fourth optical value beingdifferent from the third optical value.
 5. The electronic device ofclaim 4, wherein the third optical value includes at least one of aluminance value or an illuminance value generated based on the firstimage signal, and wherein the fourth optical value includes a colortemperature value generated based on the first image signal.
 6. Theelectronic device of claim 1, wherein the first condition is that thefirst optical value is equal to or greater than a threshold value. 7.The electronic device of claim 1, wherein the processor is furtherconfigured to, based on the first optical value not satisfying the firstcondition, correct the first image signal by using an optical valuegenerated based on the first image signal.
 8. The electronic device ofclaim 7, wherein the optical value generated based on the first imagesignal includes a color temperature value generated based on the firstimage signal.
 9. The electronic device of claim 4, wherein the firstcondition is that a difference between the first optical value and thethird optical value is less than a threshold value.
 10. The electronicdevice of claim 9, wherein the processor is further configured to, basedon the difference between the first optical value and the third opticalvalue being equal to or greater than the threshold value, correct thefirst image signal by using an optical value generated based on thefirst image signal.
 11. The electronic device of claim 1, wherein thedisplay is further configured to output an image generated based on thecorrected first image signal.
 12. The electronic device of claim 1,wherein the processor is further configured to output a third imagesignal by performing auto white balance on the first image signal, andwherein the processor is further configured to, based on the firstoptical value satisfying a second condition, output a fourth imagesignal obtained by correcting the third image signal using the secondoptical value.
 13. The electronic device of claim 12, wherein thedisplay is further configured to output an image generated based on atleast one of the third image signal or the fourth image signal.
 14. Anelectronic device, comprising: a first image sensor configured to outputa first image signal based on sensing a first light incident to a frontsurface of the electronic device; a second image sensor configured tooutput a second image signal based on sensing a second light incident toa rear surface of the electronic device; a processor configured toreceive the first image signal and the second image signal; and adisplay disposed on the front surface and configured to output an imagegenerated based on the first image signal, wherein the processor isfurther configured to generate a first color temperature value based onthe second image signal, and based on a first condition being satisfied,correct the first image signal by using the first color temperaturevalue.
 15. The electronic device of claim 14, wherein the display isfurther configured to cover the first image sensor.
 16. The electronicdevice of claim 14, wherein the processor is further configured togenerate a second color temperature value based on the first imagesignal, and generate at least one of a luminance value or an illuminancevalue based on the second image signal.
 17. The electronic device ofclaim 16, wherein the first condition is that the at least one of theluminance value or the illuminance value is equal to or greater than athreshold value.
 18. The electronic device of claim 16, wherein theprocessor is further configured to, based on the first condition notbeing satisfied, correct the first image signal by using the secondcolor temperature value.
 19. The electronic device of claim 14, whereinthe display is further configured to output an image generated based onthe corrected first image signal.
 20. An electronic device comprising: adisplay; a first camera module including a first image sensor that isconfigured to output a first image signal based on sensing a first lightpassing through the display; a second camera module including a secondimage sensor that is configured to output a second image signal based onsensing a second light that does not pass through the display; and anapplication processor disposed separately from the first camera moduleand the second camera module, the application processor including animage signal processor, wherein the image signal processor is configuredto: receive the first image signal from the first camera module througha first camera serial interface, receive the second image signal fromthe second camera module through a second camera serial interface,generate a first color temperature value based on the first imagesignal, generate, based on the second image signal, at least one of aluminance value or an illuminance value, and a second color temperaturevalue, and correct the first image signal based on the second colortemperature value based on the at least one of the luminance value orthe luminance value being equal to or greater than a threshold value,and correct the first image signal based on the first color temperaturevalue based on the at least one of the luminance value or theilluminance value being less than the threshold value.